Skip to content

feat: serialize ASOF join plans - #23832

Draft
Xuanwo wants to merge 19 commits into
apache:mainfrom
Xuanwo:xuanwo/asof-proto
Draft

feat: serialize ASOF join plans#23832
Xuanwo wants to merge 19 commits into
apache:mainfrom
Xuanwo:xuanwo/asof-proto

Conversation

@Xuanwo

@Xuanwo Xuanwo commented Jul 23, 2026 •
edited
Loading

Copy link
Copy Markdown
Member

Which issue does this PR close?

Rationale for this change

This is the serialization layer of the ASOF JOIN stack. It gives logical and
physical ASOF plans explicit protobuf representations without coupling wire
format review to SQL or DataFrame APIs.

GitHub shows the cumulative stack because fork branches cannot be selected as
upstream PR bases. The isolated diff for this layer is:
Xuanwo/datafusion@xuanwo/asof-logical...xuanwo/asof-proto

What changes are included in this PR?

  • Add protobuf messages and enum values for logical and physical ASOF joins.
  • Encode and decode equality keys, ordered match expressions, match direction,
    join constraint, and right output indices.
  • Implement physical serialization through ExecutionPlan::try_to_proto and
    AsOfJoinExec::try_from_proto, following the current self-serializing
    execution-plan pattern.
  • Regenerate prost and pbjson sources with the repository generator.
  • Add logical round trips for all four match directions and a physical
    AsOfJoinExec round trip.

Are these changes tested?

Yes:

  • ./datafusion/proto-models/regen.sh
  • cargo fmt --all
  • cargo clippy --all-targets --all-features -- -D warnings
  • cargo test -p datafusion-proto roundtrip_asof_join --all-features

Are there any user-facing changes?

Logical and physical ASOF join plans can be serialized through
datafusion-proto. The additions use new messages and append-only oneof/enum
tags, so existing wire tags are not reused. Generated public Rust enums gain new
variants, however, so downstream exhaustive matches must add arms; this is a
Rust source-compatibility break even though the wire additions are compatible.

This PR should be reviewed and merged after #23829. After its dependencies
squash-merge, this branch will be refreshed from main without rewriting its
published history.

@github-actions github-actions Bot added sql SQL Planner logical-expr Logical plan and expressions optimizer Optimizer rules core Core DataFusion crate substrait Changes to the substrait crate proto Related to proto crate physical-plan Changes to the physical-plan crate labels Jul 23, 2026
@Xuanwo Xuanwo mentioned this pull request Jul 23, 2026
Open
@github-actions

github-actions Bot commented Jul 23, 2026 •
edited
Loading

Copy link
Copy Markdown

Thank you for opening this pull request!

Reviewer note: cargo-semver-checks reported the current version number is not SemVer-compatible with the changes in this pull request (compared against the base branch).

Details 
     Cloning apache/main
    Building datafusion v54.1.0 (current)
       Built [ 106.886s] (current)
     Parsing datafusion v54.1.0 (current)
      Parsed [   0.034s] (current)
    Building datafusion v54.1.0 (baseline)
       Built [ 102.679s] (baseline)
     Parsing datafusion v54.1.0 (baseline)
      Parsed [   0.035s] (baseline)
    Checking datafusion v54.1.0 -> v54.1.0 (no change; assume patch)
     Checked [   0.616s] 223 checks: 223 pass, 30 skip
     Summary no semver update required
    Finished [ 212.235s] datafusion
    Building datafusion-expr v54.1.0 (current)
       Built [  26.574s] (current)
     Parsing datafusion-expr v54.1.0 (current)
      Parsed [   0.075s] (current)
    Building datafusion-expr v54.1.0 (baseline)
       Built [  26.808s] (baseline)
     Parsing datafusion-expr v54.1.0 (baseline)
      Parsed [   0.078s] (baseline)
    Checking datafusion-expr v54.1.0 -> v54.1.0 (no change; assume patch)
     Checked [   1.273s] 223 checks: 222 pass, 1 fail, 0 warn, 30 skip

--- failure enum_variant_added: enum variant added on exhaustive enum ---

Description:
A publicly-visible enum without #[non_exhaustive] has a new variant.
        ref: https://doc.rust-lang.org/cargo/reference/semver.html#enum-variant-new
       impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.49.0/src/lints/enum_variant_added.ron

Failed in:
  variant LogicalPlan:AsOfJoin in /home/runner/work/datafusion/datafusion/datafusion/expr/src/logical_plan/plan.rs:301
  variant LogicalPlan:AsOfJoin in /home/runner/work/datafusion/datafusion/datafusion/expr/src/logical_plan/plan.rs:301

     Summary semver requires new major version: 1 major and 0 minor checks failed
    Finished [  56.101s] datafusion-expr
    Building datafusion-optimizer v54.1.0 (current)
       Built [  26.903s] (current)
     Parsing datafusion-optimizer v54.1.0 (current)
      Parsed [   0.031s] (current)
    Building datafusion-optimizer v54.1.0 (baseline)
       Built [  26.516s] (baseline)
     Parsing datafusion-optimizer v54.1.0 (baseline)
      Parsed [   0.032s] (baseline)
    Checking datafusion-optimizer v54.1.0 -> v54.1.0 (no change; assume patch)
     Checked [   0.170s] 223 checks: 223 pass, 30 skip
     Summary no semver update required
    Finished [  54.472s] datafusion-optimizer
    Building datafusion-physical-plan v54.1.0 (current)
       Built [  36.299s] (current)
     Parsing datafusion-physical-plan v54.1.0 (current)
      Parsed [   0.145s] (current)
    Building datafusion-physical-plan v54.1.0 (baseline)
       Built [  36.148s] (baseline)
     Parsing datafusion-physical-plan v54.1.0 (baseline)
      Parsed [   0.142s] (baseline)
    Checking datafusion-physical-plan v54.1.0 -> v54.1.0 (no change; assume patch)
     Checked [   0.642s] 223 checks: 223 pass, 30 skip
     Summary no semver update required
    Finished [  74.928s] datafusion-physical-plan
    Building datafusion-proto v54.1.0 (current)
       Built [  59.415s] (current)
     Parsing datafusion-proto v54.1.0 (current)
      Parsed [   0.018s] (current)
    Building datafusion-proto v54.1.0 (baseline)
       Built [  59.529s] (baseline)
     Parsing datafusion-proto v54.1.0 (baseline)
      Parsed [   0.019s] (baseline)
    Checking datafusion-proto v54.1.0 -> v54.1.0 (no change; assume patch)
     Checked [   0.239s] 223 checks: 223 pass, 30 skip
     Summary no semver update required
    Finished [ 120.741s] datafusion-proto
    Building datafusion-proto-models v54.1.0 (current)
       Built [  24.620s] (current)
     Parsing datafusion-proto-models v54.1.0 (current)
      Parsed [   0.130s] (current)
    Building datafusion-proto-models v54.1.0 (baseline)
       Built [  24.728s] (baseline)
     Parsing datafusion-proto-models v54.1.0 (baseline)
      Parsed [   0.131s] (baseline)
    Checking datafusion-proto-models v54.1.0 -> v54.1.0 (no change; assume patch)
     Checked [   1.774s] 223 checks: 222 pass, 1 fail, 0 warn, 30 skip

--- failure enum_variant_added: enum variant added on exhaustive enum ---

Description:
A publicly-visible enum without #[non_exhaustive] has a new variant.
        ref: https://doc.rust-lang.org/cargo/reference/semver.html#enum-variant-new
       impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.49.0/src/lints/enum_variant_added.ron

Failed in:
  variant LogicalPlanType:AsOfJoin in /home/runner/work/datafusion/datafusion/datafusion/proto-models/src/generated/prost.rs:83
  variant LogicalPlanType:AsOfJoin in /home/runner/work/datafusion/datafusion/datafusion/proto-models/src/generated/prost.rs:83
  variant PhysicalPlanType:AsOfJoin in /home/runner/work/datafusion/datafusion/datafusion/proto-models/src/generated/prost.rs:1407
  variant PhysicalPlanType:AsOfJoin in /home/runner/work/datafusion/datafusion/datafusion/proto-models/src/generated/prost.rs:1407

     Summary semver requires new major version: 1 major and 0 minor checks failed
    Finished [  52.409s] datafusion-proto-models
    Building datafusion-sql v54.1.0 (current)
       Built [  41.600s] (current)
     Parsing datafusion-sql v54.1.0 (current)
      Parsed [   0.032s] (current)
    Building datafusion-sql v54.1.0 (baseline)
       Built [  41.511s] (baseline)
     Parsing datafusion-sql v54.1.0 (baseline)
      Parsed [   0.032s] (baseline)
    Checking datafusion-sql v54.1.0 -> v54.1.0 (no change; assume patch)
     Checked [   0.232s] 223 checks: 223 pass, 30 skip
     Summary no semver update required
    Finished [  85.024s] datafusion-sql
    Building datafusion-substrait v54.1.0 (current)
       Built [ 343.431s] (current)
     Parsing datafusion-substrait v54.1.0 (current)
      Parsed [   0.018s] (current)
    Building datafusion-substrait v54.1.0 (baseline)
       Built [ 353.310s] (baseline)
     Parsing datafusion-substrait v54.1.0 (baseline)
      Parsed [   0.017s] (baseline)
    Checking datafusion-substrait v54.1.0 -> v54.1.0 (no change; assume patch)
     Checked [   0.206s] 223 checks: 223 pass, 30 skip
     Summary no semver update required
    Finished [ 699.363s] datafusion-substrait

@github-actions github-actions Bot added the auto detected api change Auto detected API change label Jul 23, 2026
@codecov-commenter

codecov-commenter commented Jul 23, 2026 •
edited
Loading

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 61.56236% with 866 lines in your changes missing coverage. Please review.
✅ Project coverage is 80.56%. Comparing base (e8a65f2) to head (aaae20f).
⚠️ Report is 10 commits behind head on main.

Files with missing lines Patch % Lines
datafusion/proto-models/src/generated/pbjson.rs 0.00% 341 Missing ⚠️
datafusion/physical-plan/src/joins/asof_join.rs 86.11% 119 Missing and 74 partials ⚠️
datafusion/expr/src/logical_plan/plan.rs 38.70% 100 Missing and 14 partials ⚠️
datafusion/core/src/physical_planner.rs 0.00% 43 Missing ⚠️
datafusion/expr/src/logical_plan/tree_node.rs 0.00% 43 Missing ⚠️
datafusion/proto/src/logical_plan/mod.rs 62.06% 20 Missing and 13 partials ⚠️
datafusion/optimizer/src/analyzer/type_coercion.rs 0.00% 28 Missing ⚠️
datafusion/optimizer/src/push_down_filter.rs 0.00% 24 Missing ⚠️
datafusion/proto-models/src/generated/prost.rs 0.00% 17 Missing ⚠️
datafusion/expr/src/logical_plan/builder.rs 82.45% 1 Missing and 9 partials ⚠️
... and 4 more
Additional details and impacted files 
@@            Coverage Diff             @@
##             main   #23832      +/-   ##
==========================================
- Coverage   80.66%   80.56%   -0.10%     
==========================================
  Files        1095     1096       +1     
  Lines      372294   374784    +2490     
  Branches   372294   374784    +2490     
==========================================
+ Hits       300324   301959    +1635     
- Misses      54055    54772     +717     
- Partials    17915    18053     +138

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Xuanwo added 12 commits July 27, 2026 16:22
@Xuanwo
Merge remote-tracking branch 'origin/main' into xuanwo/asof-physical
73fd599
@Xuanwo
feat: broadcast ASOF right input
43110a3
@Xuanwo
Merge branch 'xuanwo/asof-physical' into xuanwo/asof-logical
a15b693 
# Conflicts:
#	datafusion/expr/src/logical_plan/plan.rs
@Xuanwo
Merge branch 'xuanwo/asof-logical' into xuanwo/asof-proto
88a46e1 
# Conflicts:
#	datafusion/proto/src/physical_plan/mod.rs
@Xuanwo
fix: remove obsolete ASOF proto import
9556ed3
@Xuanwo
fix: account shared ASOF build buffers once
339d785
@Xuanwo
Merge branch 'xuanwo/asof-physical' into xuanwo/asof-logical
a71c19f
@Xuanwo
fix: align ASOF logical contracts
3a4f995
@Xuanwo
Merge branch 'xuanwo/asof-logical' into xuanwo/asof-proto
ce2fbc4
@Xuanwo
perf: compare ASOF keys without scalar materialization
34552d1
@Xuanwo
Merge branch 'xuanwo/asof-physical' into xuanwo/asof-logical
c967002
@Xuanwo
Merge branch 'xuanwo/asof-logical' into xuanwo/asof-proto
aaae20f
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

auto detected api change Auto detected API change core Core DataFusion crate logical-expr Logical plan and expressions optimizer Optimizer rules physical-plan Changes to the physical-plan crate proto Related to proto crate sql SQL Planner substrait Changes to the substrait crate

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@Xuanwo @codecov-commenter